Inter-cluster communications technique for event and health status communications

ABSTRACT

Communication between clusters of processing nodes is initiated by designating a link layer address and a target node name in a remote cluster. The link layer address is transmitted to the target node via a regular network transmission. The link layer address is stored locally in each cluster and the nodes in each cluster are signaled that a configuration change has been made, so that gateway nodes in the clusters can obtain the link layer address to use for subsequent communications with the other cluster, such as event notifications of node status changes.

BACKGROUND

1. Field of the Invention

The present invention is related to communications between nodes indifferent clusters in large-scale distributed computer systems, and morespecifically to a technique for inter-cluster communication that sharesa link layer address between clusters.

2. Description of Related Art

In large-scale distributed computer systems, such as those usingdistributed software models to perform tasks, multiple nodes provideindependent execution of sub-tasks. The nodes are typically organizedinto clusters in order to perform specific tasks and/or as a physicalorganization of a processing system, for example, clustering byprocessor affinity. In order to keep such a system operational, andfurther, to provide indication of events occurring at one node thateither require a reaction from another node or indicate to the othernode that either an erroneous operating condition has occurred, or thata phase of processing is complete. In particular, event notification andevent logging are operations used to indicate system health to systemadministrators or software applications, including operating systemscomponents.

Health monitoring techniques employed in distributed processing systemsperform an important function in that connections to other nodes must bereliable and all of the active nodes that have been assigned tasks needto perform those tasks in order to ensure that the totality of theprocessing requirements are met, and in a timely fashion. The health ofa node-based distributed processing system is typically monitored by: 1)a heartbeat messaging system, which passes messages between the nodesand a central monitoring component; and 2) an event notification systemthat signals interested nodes when events occur on other nodes. Eventnotification systems in node-based distributed processing systemstypically require an interested application (a consumer) to register toreceive event notifications either with a centralized event manager, orwith the processes or objects that generate the events (an eventproducer). However, the event communications and health statusmonitoring are typically performed only among nodes in a single cluster.

BRIEF SUMMARY

The invention provides for communications between nodes in differentclusters using methods that are embodied in computer-performed methods.The methods provide communications between clusters of nodes that maynot share the same network by sharing a link layer address among thenodes in a cluster by storing the link layer address in a persistentstorage.

The link layer address is specified along with a target node name in aremote cluster, generally by an administrator setting up the clusterconfigurations. A local node receiving the link layer address stores thelink layer address in local storage accessible by the nodes in the localcluster and sends the local nodes a configuration change message so thatthe local nodes can retrieve the link layer address to use forsubsequent communications with nodes in the remote cluster. The targetnode name is used to communicate the link layer address to the remotecluster. Upon receiving the link layer address, the target node storesthe link layer address in a storage accessible by the nodes in theremote cluster and notifies the nodes that the configuration change hastaken place, so that a gateway node(s) in the remote cluster canretrieve the link layer address and use the link layer address tocommunicate with the gateway node(s) in the other cluster.

The foregoing and other objectives, features, and advantages of theinvention will be apparent from the following, more particular,description of the preferred embodiments of the invention, asillustrated in the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asa preferred mode of use, further objectives, and advantages thereof,will best be understood by reference to the following detaileddescription of the invention when read in conjunction with theaccompanying Figures, wherein like reference numerals indicate likecomponents, and:

FIG. 1 is a block diagram of a cluster within a distributed computersystem in which techniques according to an embodiment of the presentinvention are practiced.

FIG. 2 is a block diagram of a distributed computer system in whichtechniques according to an embodiment of the present invention arepracticed.

FIG. 3 is a pictorial diagram depicting communication between nodes of adistributed computer system in accordance with an embodiment of thepresent invention.

FIG. 4 is a pictorial diagram depicting information flow in adistributed computer system in accordance with an embodiment of thepresent invention.

FIG. 5 is a flowchart of a method in accordance with an embodiment ofthe present invention.

DETAILED DESCRIPTION

The present invention encompasses techniques for communication betweennodes in different clusters of nodes within distributed computingsystems. In particular, embodiments of the invention provide forcommunication of node status changes between different clusters that maybe in different geographical locations and on different local areanetworks (LANs), and communication of other inter-node messaging, suchas event messages. Gateway node(s) in each cluster communicate statuschanges such as node down/up events for their cluster to the remotecluster(s), providing a low bandwidth usage status monitoring.

Referring now to FIG. 1, a cluster 20 within a distributed computersystem in accordance with an embodiment of the present invention isshown. A first physical processing node 10A includes a processor core 12coupled to a memory 14 that stores program instructions for execution byprocessor 12. The program instructions include program instructionsforming computer program products in accordance with embodiments of theinvention that provide set up of communications and subsequent messagingbetween either physical processing nodes 10A-10D, or virtual processingnodes partitioned within the depicted computer system as will bedescribed in further detail below, and remote clusters. Processing node10A also includes a network interface (NWI) 16 that couples processingnode 10A to a wired, wireless or hybrid network, which may be astandardized network such as Ethernet, or a proprietary network orinterconnect bus. Other processing nodes 10B-10D are of identicalconstruction in the exemplary embodiment, but embodiments of theinvention may be practiced in asymmetric distributed systems havingnodes with differing features. Although only four compute nodes 10A-10Dare illustrated in cluster 20, a distributed computer system inaccordance with an embodiment of the present invention will generallyinclude a large number of compute nodes per cluster and connected viaone or more networks. The distributed computer system of FIG. 1 alsoincludes other resources such as I/O devices 19, including graphicaldisplay devices, printers, scanners, keyboards, mice, which may becoupled to the network or one of nodes 10A-10D via workstation computersthat provide a user interface to administrative personnel and otherusers. Nodes 10A-10D are also coupled to storage devices 18, for storingand retrieving data and program instructions, such as storing computerprogram products in accordance with an embodiment of the invention.

Referring now to FIG. 2, a distributed computer system in accordancewith an embodiment of the present invention is shown. The depictedcomputer system includes a plurality of clusters 20A, 20B and 20C, thateach include a plurality of processing nodes 10A, 10B in cluster 20A,nodes 10E, 10F in cluster 20B and nodes 10G, 10H in cluster 20C. Ingeneral, clusters 20A, 20B and 20C may be geographically distant fromeach other, which may be the reason for partitioning the particularnodes into the clusters, since communication across the remoteinterfaces will generally be slower and/or more costly. However, thereare other reasons for partitioning systems into clusters, such as forcommunications traffic management, and the present invention is notlimited as to the geographical arrangement of the clusters. Each ofclusters 20A, 20B and 20C, has at least one corresponding storage device18A, 18B and 18C that is readable by all of the nodes in the cluster. Inthe present invention, the local storage device is used to shareconfiguration between the nodes in a cluster, including the link addressthat is shared between two or more clusters to enable remotecommunication between nodes. Clusters 20A, 20B and 20C may not share atop level network such as an Internet Protocol (IP) network, andtherefore an address such as an IP address may not uniquely identify asingle node or other physical or virtual device. Therefore, as will beexplained in further detail below, a node name of a target node is usedto identify a remote node to which a link layer address will be passed,so that the node name can be resolved on the local network prior to setup of lower latency communication via the link layer address.

Referring now to FIG. 3, communication between multiple nodes 10 of thedistributed computer system of FIG. 2 is shown. As mentioned above,nodes 10 may correspond exactly on a one-to-one basis with processingnodes 10A-10H (and other nodes) of FIGS. 1-2, or nodes 10 may bepartitioned in a different manner as virtual processing nodes, and nodes10 can be located in different clusters without requiring communicationwith a particular interface to a remote cluster. For example, a singlenode 10 may have exclusive use of multiple processing nodes, e.g. nodes10A-10B, and result in a system having a greater number of virtual nodesthan processing nodes, or alternatively, multiple nodes 10 may beimplemented on a single processing node, e.g., node 10A. In theexemplary embodiment, each of nodes 10 represents at least one operatingsystem image and one or more applications executing within the operatingsystem image. In general, the entire system as depicted may execute asingle application, but sub-tasks within the application are apportionedto the various nodes 10, which may be identical sub-tasks or differentsub-tasks. Messages passed between nodes 10, include events, as well asdata and program code transmissions, and messages as needed to supportfeatures of embodiments of the present invention, such as gossiping,event messages and other communications that are transmitted to andreceived from nodes in remote clusters. A separate physical network maybe provided for administrative tasks such as event notification andheartbeat messaging, or the same physical networks may be used. Theclusters are generally a group of virtual or physical processing nodesorganized to perform a particular task or group of tasks, e.g., for aparticular customer. As illustrated, communication between nodes in acluster may be accomplished by direct node to node communications 22 orby next-neighbor communications 24 in which nodes 10 pass along messagesto other nodes 10. Another alternative is to use a central facility forinter-node communication. However, use of a central facility causes acommunications bottleneck, for which reason, among others, passing of alink layer address by which nodes 10 can communicate directly withremote nodes is desirable. By allowing nodes 10 to provide interfaces toother nodes (including remote nodes) directly, communications trafficbetween nodes can be programmatically independent of what network paththe communications will take. (Whether a bottleneck actually existsbetween two clusters then becomes a question of what node-accessiblecommunications resources are actually present between the clusters totransmit and receive the inter-cluster communications.) In general, thelink between clusters could be used for any communication, but in thedepicted embodiment, gateway nodes in each cluster generally share onlysignificant events that do not require much bandwidth to communicate,such as node status change events.

Referring now to FIG. 4, information flow in a computer system inaccordance with an embodiment of the invention is shown. Two clusters40A and 40B are shown, each containing corresponding physical or virtualprocessing nodes 30A-30B and 30C-30D, respectively. Within cluster 40A,nodes 30A-30B communicate with each other directly to exchange gossipingmessages, and other information, and communicate with an instance of anAutonomic Health Advisor File System (AHAFS) 32A to transmit and receiveevent messages. Local persistent storage 34A is provided to store localshared data between nodes in cluster 40A, including the link address andother shared information as described above. Within cluster 40B, nodes30C-30D, also communicate with each other, with an AHAFS 32B instanceand have access to a local persistent storage 34B. Link 36A, 36Brepresents a communications link through which gateway nodes 30A and 30Cmay exchange messages such as node status change event messages, via theshared link layer address that corresponds to link 36A, 36B. Link 36A,36B does not correspond to a physical entity, but rather is used in thediagram to depict a connection that provides for inter-communicationbetween gateway nodes 30A, 30C, which share information received fromother components such as AHAFS instances 32A-32B in clusters 40A-40B.Link 36A, 36B exists by observing messages directed at the unique linklayer address, e.g., a media access control (MAC) layer address that isspecified for communications between gateway nodes 30A and 30C inclusters 40A-40B. One or more unique link layer addresses may bespecified for each pair of clusters, and optionally a unique address maybe specified for each communication direction of link 36A, 36B. Further,while only one gateway node 30A, 30C is appointed for each cluster 40A,40B in the depicted embodiment, two gateway nodes may be used forredundancy and the number of nodes that are used as gateway nodes is nota limitation of the present invention.

Referring now to FIG. 5, a method in accordance with an embodiment ofthe present invention is depicted in a flowchart. A user (generally anadministrator) accessing a node at a local cluster specifies the linklayer address, which in the example is a multicast address, but whichcould alternatively be a unicast address, and also specifies a targetnode in the remote cluster (step 50). The node that receives the userinput stores the link address in local shared storage (step 51) andbroadcasts a configuration change message to other nodes in the localcluster (step 52). The node also transmits the link address to thetarget node using a network layer communication such as TCP/IP and usingthe target node name (step 53). The target node receives the link layeraddress transmission and stores the link layer address in local sharedstorage (step 54). The target node broadcasts a configuration changemessage to other nodes in the remote cluster (step 55). In bothclusters, once the nodes receive the configuration change message, thegateway nodes brand their interfaces with the link layer address toallow direct communication with the gateway node(s) in the other cluster(step 56). Nodes can then gossip, signal events and perform othercommunications involving nodes in the other clusters using the linklayer address (step 57).

Once the link is established, communications such as node up/down eventsassociated with health status monitoring as disclosed in U.S. patentapplication “ENDPOINT-TO-ENDPOINT COMMUNICATIONS STATUS MONITORING”,Ser. No. 12/959,556, filed on Dec. 3, 2010 and which is incorporatedherein by reference, can be extended to track communications betweenboth local and remote nodes to determine node status health. Further,the health of the link between the clusters can be monitored byobserving whether the health status of any of the remote nodes is alive,in which case the link is active. If all of the remote nodes areobserved as “dead”, then the link is dead. Link status is reported byone of the nodes in each cluster via the AHAFS framework as an event(Link UP/Link Down events). Details of the AHAFS framework are disclosedin U.S. Patent Application Publication 200901991051, published on Aug.6, 2009, and which is incorporated herein by reference.

As noted above, the present invention may be embodied as a system,method, and/or a computer program product. A computer program productmay be embodied in firmware, an image in system memory or anothermemory/cache, stored on a fixed or re-writable media such as an opticaldisc having computer-readable code stored thereon. Any combination ofone or more computer readable medium(s) may be used to store the programinstructions in accordance with an embodiment of the invention. Thecomputer readable medium may be a computer readable signal medium or acomputer readable storage medium. A computer readable storage medium maybe, for example, but not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus, ordevice, or any suitable combination of the foregoing. More specificexamples (a non-exhaustive list) of the computer readable storage mediumwould include the following: an electrical connection having one or morewires, a portable computer diskette, a hard disk, a random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), an optical fiber, a portable compactdisc read-only memory (CD-ROM), an optical storage device, a magneticstorage device, or any suitable combination of the foregoing.

In the context of the present application, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device. A computer readable signal medium may include apropagated data signal with computer readable program code embodiedtherein, for example, in baseband or as part of a carrier wave. Such apropagated signal may take any of a variety of forms, including, but notlimited to, electro-magnetic, optical, or any suitable combinationthereof. A computer readable signal medium may be any computer readablemedium that is not a computer readable storage medium and that cancommunicate, propagate, or transport a program for use by or inconnection with an instruction execution system, apparatus, or device.Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. Further, while the illustrative embodiment isdirected to an AIX (AIX is a trademark of IBM) or other type of UNIXoperating system (UNIX is a trademark of The Open Group), in which theevent notification is provided by a mountable file system provided as akernel extension, it is understood that the techniques of the presentinvention can be applied in event monitoring systems executing under anyoperating system and can be implemented using proprietary orstandardized signaling interfaces as an alternative to the file systeminterface provided in the depicted embodiments.

While the invention has been particularly shown and described withreference to the preferred embodiments thereof, it will be understood bythose skilled in the art that the foregoing and other changes in formand details may be made therein without departing from the spirit andscope of the invention.

1. A method for communication within a networked computer systemincluding multiple processing clusters, the clusters comprising aplurality of physical or virtual processing modes, the methodcomprising: designating at least one target node in a first one of theprocessing clusters; identifying the at least one target node to a givenone of the processing nodes in a second one of the processing clusters;providing a link layer address to use for communication between thefirst and second processing cluster; transmitting a message from thegiven processing node to the target node by identifying the target nodeusing the target node name, wherein the message contains the link layeraddress; and subsequently communicating between at least one firstgateway node in the first cluster and at least one second gateway nodein the second cluster using the link layer address, whereby the at leastone first gateway node and the at least one second gateway nodecommunicate independent of whether the first and second processingclusters are on the same network.
 2. The method of claim 1, furthercomprising: responsive to the designating and the identifying, storingthe link layer address in a first shared storage accessible by theprocessing nodes in the second cluster; and notifying first other onesof the processing nodes in the second cluster of a configuration change,whereby the first other nodes are notified that the link layer addressis available in the first shared storage, to enable the at least secondgateway node to communicate with the first cluster.
 3. The method ofclaim 2, further comprising: responsive to the transmitting, receivingthe message at the target node; responsive to the receiving, storing thelink layer address in a second shared storage accessible by theprocessing nodes in the first cluster; and notifying second other onesof the processing nodes in the second cluster of the configurationchange, whereby the at least one first gateway node is notified that thelink layer address is available to enable communications with the secondcluster.
 4. The method of claim 1, further comprising: responsive to thetransmitting, receiving the message at the target node; responsive tothe receiving, storing the link layer address in a second shared storageaccessible by the processing nodes in the first cluster; and notifyingsecond other ones of the processing nodes in the second cluster of theconfiguration change, whereby the at least one first gateway node isnotified that the link layer address is available to enablecommunications with the second cluster.
 5. The method of claim 1,wherein the designating designates at least two target nodes to provideredundant communication of the link address between the first and thesecond clusters.
 6. The method of claim 1, wherein the subsequentlycommunicating comprises transmitting and receiving status change eventmessages between the first and second clusters.
 7. The method of claim6, wherein the subsequently communicating comprises transmitting andreceiving node status change messages between the first and the secondclusters.